feat(appkit): migrate analytics to the modular @databricks/sdk-* - #562

What

Migrates the analytics stack (SQLWarehouseConnector + type-generator) off the legacy monolithic @databricks/sdk-experimental onto the new modular per-service @databricks/sdk-* SDK (v0.46.0, ESM-only), behind the existing packages/shared/src/workspace-client/ facade seam.

Analytics doesn't touch the SDK directly — it delegates to SQLWarehouseConnector, which uses two services: warehouses and statementExecution. Both move together in this increment; every other service (files/genie/jobs/serving/currentUser/apiClient) still routes through the legacy client behind the same facade (the seam's designed incremental state).

How

• New boundary module packages/shared/src/workspace-client/modular.ts — the sole importer of @databricks/sdk-* (oxlint no-restricted-imports boundary), mirroring legacy.ts. Builds per-service WarehousesClient / StatementExecutionClient; maps wrapper options → ClientOptions (host scheme-normalization, credential resolution, and a httpClient transport that stamps the User-Agent).
• Connector + type-generator rewritten to the modular API: method renames (getStatement→getStatementResult, getStatementResultChunkN→getResultData), camelCase response model, CallOptions { signal } for cancellation.
• pnpm patch (patches/@databricks__sdk-statementexecution@0.46.0.patch) restores the undocumented Reyden attachment response field that the SDK's generated unmarshal transform would otherwise strip. Version-pinned — re-apply on any SDK bump; an upstream fix is preferred long-term.

Authentication (the foundation for migrating the remaining services)

Auth resolution is centralized in one service-agnostic function, mapToClientOptions(opts) in modular.ts, which produces the ClientOptions { host, credentials, httpClient } that every modular client constructor accepts. Warehouses and statementExecution build through it today; each remaining service will build through the same function, so the rest of the migration needs no per-service auth work (the credentials are generic bearer TokenCredentials, not service-scoped).

Resolution order — restores the legacy SDK's behavior for the modes AppKit uses:

• explicit token (asUser OBO) → PAT, guarded by token !== undefined so an empty/invalid OBO token fails loudly instead of silently falling through to the service principal (privilege-escalation guard);
• profile → profile file;
• otherwise the service principal from the environment — DATABRICKS_CLIENT_ID + DATABRICKS_CLIENT_SECRET → OAuth M2M (what the Databricks Apps runtime injects), else DATABRICKS_TOKEN → PAT, else the SDK's profile-based default chain.

Why resolve the SP explicitly rather than lean on the SDK's default chain: the default chain does read the DATABRICKS_* env vars, but its M2M strategy performs OAuth token-endpoint discovery against the raw DATABRICKS_HOST, which the Apps runtime injects scheme-less (e.g. e2-dogfood.staging.cloud.databricks.com). Discovery then throws Invalid URL and every deployed request fails as "Warehouse readiness check failed". Resolving the SP here with the scheme-normalized host avoids that (it's the credential-side facet of the host-normalization fix below). Verified end-to-end on a deployed app.

Not a full 1-to-1 with legacy's auth surface: the modular SDK 0.46.0 supports only PAT, OAuth M2M, and OAuth U2M — there are no factories for Azure (MSI/CLI/SP), GCP, or the instance-metadata service that legacy unified-auth covered. This is behaviorally equivalent for AppKit's real environments (Databricks Apps = SP via env; local dev = profile / PAT / env) but a reduced surface for those exotic modes; noted for release notes.

Per-service OBO scopes: the auth code is uniform, but each OBO-consuming service needs its matching user_api_scopes declared in the app's databricks.yml — sql (warehouse/statement), files.files, dashboards.genie, serving.serving-endpoints. That's app configuration, not auth code.

Runtime bugs found + fixed (caught against live warehouses / a deployed app, not unit tests)

1. Host scheme (requests) — the modular SDK passes host straight to fetch; a bare DATABRICKS_HOST → Invalid URL. Fixed via normalizeHost on the request host.
2. User-Agent — setProduct("@databricks/appkit") throws (the @// aren't token chars) and client-info is process-global. AppKit dashboards filter traffic on the literal @databricks/appkit UA, so we set it on a per-client httpClient transport wrapper instead — preserving the exact string. (A pnpm patch of the validator wouldn't reach deployed apps, which is where the dashboard traffic originates.)
3. ApiError.code — the modular ApiError carries the Databricks error code on .code, not .errorCode; the connector now reads either, so the arrow disposition/format capability-rejection fallback (INLINE↔EXTERNAL_LINKS) fires again.
4. BigInt serialization — the modular SDK types row/byte counts as bigint; Reyden's cached INLINE+ARROW_STREAM result then broke JSON.stringify. Counts are coerced back to number at the connector boundary.
5. Deployed service-principal auth — the SDK's M2M strategy builds credentials from the raw scheme-less DATABRICKS_HOST the Apps runtime injects, so OAuth token discovery failed (Invalid URL) and every deployed query died as "Warehouse readiness check failed". Fixed by resolving the SP with a scheme-normalized host (see Authentication).

Testing

• Full appkit + shared suites green, including new regression tests for the .code read, bigint normalization, the env-based credential resolution (M2M-from-env, PAT-from-env, OBO-token-wins, M2M-needs-host), and the @databricks/appkit User-Agent stamping.
• pnpm -r typecheck, lint, format clean; pnpm install --frozen-lockfile green (appkit importer synced in pnpm-lock.yaml).
• Verified live against standard + Reyden serverless warehouses: JSON + arrow (INLINE attachment and EXTERNAL_LINKS), OBO (.obo.sql), warehouse auto-start, metric views.
• Verified end-to-end on a deployed Databricks App: an SP-default query and an OBO (asUser, user_api_scopes: [sql]) query both execute against the SQL warehouse.

Notes

• Metric views run entirely on the migrated services (they compile to SQL); no separate migration needed.
• Regenerated the workspace-client API docs to match the new types (included).

This pull request and its description were written by Isaac.